This script is the visual inspection scripts. 1. It takes the data file on EDI and combines it with the L1 file which has already been QAQCed. 2. Then the script checks for daily and subdaily gaps in the current file. 3. Lists the flag frequency to check if there are any NAs or any assigned the wrong flag. 4. If necessary can QAQC data already on EDI using the QAQC function 5. Creates plots 6. Writes data to new csv
For the plots, they use a function called “all_plot”. In all_plot you can specify if you want plotly plots for the current data. BEWARE if you turn on the plotly plots and try to knit the markdown it will fail! I am working on a fix. For right now you can specify which plotly plots you want on. You can also look at the plotly plots manually in each chunk by running the chunk with Use_plotly=TRUE as an argument and then at the end of the chunk output[[1]].
All files are from the link to GitHub and the source scripts are from GitHub as well.
This identifies if there are any daily data gaps in the long-term record
## [1] "2023-08-08 12:20:00 UTC" "2023-08-14 11:30:00 UTC"
## [1] "2023-08-19 04:20:00 UTC" "2023-08-21 09:30:00 UTC"
This identifies if there are any sub-daily gaps in the current record, as gaps of the long term record are found in the metadata.
The first row is the time for the first observation and then the subsequent observation. Each observation should be 10 minutes apart. The second row is the number of the record for each observation. Most of these gaps happen when we change the program on the data logger. These times will be recorded in the maintenance section of the metadata and are also noted in the maintenance log.
## [1] "2023-03-13 14:00:00 UTC" "2023-03-13 14:20:00 UTC"
## [1] 1882 0
## [1] "2023-06-07 12:00:00 UTC" "2023-06-07 12:30:00 UTC"
## [1] 2 0
## [1] "2023-06-27 11:00:00 UTC" "2023-06-27 11:20:00 UTC"
## [1] 2871 0
## [1] "2023-08-08 12:20:00 UTC" "2023-08-14 11:30:00 UTC"
## [1] 6054 6913
## [1] "2023-08-19 04:20:00 UTC" "2023-08-21 09:30:00 UTC"
## [1] 7590 7909
QAQC Historical Files: If you need to QAQC the historical file then run it through the qaqc function without start_date and end_date as NULL
This section reads in the QAQC file and then you can look at the head, tail and structure. Make sure the last row is Dec. 31 23:50 of the publishing year. There should be 85 columns unless a new one has been added.
This is information that is included in the methods file for EDI
The minimum water level for BVR was 15.6535017 and the maximum water depth was 19.6358085.
The median water level was 18.5887838 and the mean water level was 18.4561891.
Let’s look at the flag Frequency for each variable. As a reminder here are the flag codes Flag values
0: no flag
1: value removed due to maintenance and set to NA
2: negative or outlier value removed and set to NA, see Methods section for more detail on QAQC process
3: negative values set to 0 except for temperature
4: value removed due to fouling and set to NA
5: questionable value but left in the dataset
6: Values adjusted using a linear or square root function to match high-resolution CTD profiles and corrected other affected observations on the same sensor
7: missing data
## Flag_ThermistorTemp_C_1
## 0 2 7 <NA>
## 43664 93965 4633 0
## Flag_ThermistorTemp_C_2
## 0 2 7 <NA>
## 104015 33616 4631 0
## Flag_ThermistorTemp_C_3
## 0 2 7 <NA>
## 123158 14473 4631 0
## Flag_ThermistorTemp_C_4
## 0 2 7 <NA>
## 136597 1034 4631 0
## Flag_ThermistorTemp_C_5
## 0 2 7 <NA>
## 137582 48 4632 0
## Flag_ThermistorTemp_C_6
## 0 2 7 <NA>
## 137582 48 4632 0
## Flag_ThermistorTemp_C_7
## 0 2 7 <NA>
## 137582 48 4632 0
## Flag_ThermistorTemp_C_8
## 0 2 7 <NA>
## 137579 48 4635 0
## Flag_ThermistorTemp_C_9
## 0 2 7 <NA>
## 137582 48 4632 0
## Flag_ThermistorTemp_C_10
## 0 2 7 <NA>
## 137578 48 4636 0
## Flag_ThermistorTemp_C_11
## 0 2 7 <NA>
## 137583 48 4631 0
## Flag_ThermistorTemp_C_12
## 0 2 7 <NA>
## 137582 48 4632 0
## Flag_ThermistorTemp_C_13
## 0 1 2 7 <NA>
## 134740 2843 48 4631 0
## Flag_EXOTemp_C_1
## 0 1 2 7 <NA>
## 105029 51 5163 32019 0
## Flag_EXOCond_uScm_1
## 0 1 2 7 <NA>
## 105029 51 5163 32019 0
## Flag_EXOSpCond_uScm_1
## 0 1 2 7 <NA>
## 105028 51 5164 32019 0
## Flag_EXOTDS_mgL_1
## 0 1 2 7 <NA>
## 105028 51 5164 32019 0
## Flag_EXODOsat_percent_1
## 0 1 2 7 <NA>
## 104804 51 5163 32244 0
## Flag_EXODO_mgL_1
## 0 1 2 7 <NA>
## 104804 51 5163 32244 0
## Flag_EXOChla_RFU_1
## 0 1 2 3 7 <NA>
## 104880 51 5301 11 32019 0
## Flag_EXOChla_ugL_1
## 0 1 2 3 6 7 <NA>
## 71833 51 5286 1449 31624 32019 0
## Flag_EXOBGAPC_RFU_1
## 0 1 2 3 7 <NA>
## 76503 51 5170 28519 32019 0
## Flag_EXOBGAPC_ugL_1
## 0 1 2 3 6 7 <NA>
## 31000 51 5172 56150 17870 32019 0
## Flag_EXOfDOM_RFU_1
## 0 1 2 7 <NA>
## 105011 51 5181 32019 0
## Flag_EXOfDOM_QSU_1
## 0 1 2 7 <NA>
## 105011 51 5181 32019 0
## Flag_EXOPressure_psi_1
## 0 1 2 7 <NA>
## 104986 51 5206 32019 0
## Flag_EXODepth_m_1
## 0 1 2 7 <NA>
## 104985 51 5207 32019 0
## Flag_EXOBattery_V_1
## 0 1 2 7 <NA>
## 105027 51 5163 32021 0
## Flag_EXOCablepower_V_1
## 0 1 2 7 <NA>
## 105029 51 5163 32019 0
## Flag_EXOWiper_V_1
## 0 1 2 7 <NA>
## 71916 16341 5163 48842 0
## Flag_EXOTemp_C_9
## 0 1 2 7 <NA>
## 110501 619 4031 27111 0
## Flag_EXOCond_uScm_9
## 0 1 2 7 <NA>
## 110308 619 4224 27111 0
## Flag_EXOSpCond_uScm_9
## 0 1 2 7 <NA>
## 110308 619 4224 27111 0
## Flag_EXOTDS_mgL_9
## 0 1 2 7 <NA>
## 110308 619 4224 27111 0
## Flag_EXODOsat_percent_9
## 0 1 2 7 <NA>
## 110294 619 4033 27316 0
## Flag_EXODO_mgL_9
## 0 1 2 7 <NA>
## 110296 619 4031 27316 0
## Flag_EXOfDOM_RFU_9
## 0 1 2 7 <NA>
## 109391 619 5141 27111 0
## Flag_EXOfDOM_QSU_9
## 0 1 2 7 <NA>
## 109391 619 5141 27111 0
## Flag_EXOPressure_psi_9
## 0 1 2 7 <NA>
## 110501 619 4031 27111 0
## Flag_EXODepth_m_9
## 0 1 2 7 <NA>
## 110501 619 4031 27111 0
## Flag_EXOBattery_V_9
## 0 1 2 7 <NA>
## 110501 619 4031 27111 0
## Flag_EXOCablepower_V_9
## 0 1 2 7 <NA>
## 110501 619 4031 27111 0
## Flag_EXOWiper_V_9
## 0 1 2 7 <NA>
## 105936 5184 4031 27111 0
## Flag_LvlPressure_psi_13
## 0 2 7 <NA>
## 137583 48 4631 0
## Flag_LvlTemp_C_13
## 0 2 7 <NA>
## 137584 47 4631 0
For the plots, they use a function called “all_plot”. In all_plot you can specify if you want plotly plots for the current data. BEWARE if you turn on the plotly plots and try to knit the markdown it will fail! I am working on a fix. For right now you can specify which plotly plots you want on. You can also look at the plotly plots manually in each chunk by running the chunk with Use_plotly=TRUE as an argument and then at the end of the chunk output[[1]].
## Warning: Removed 32252 rows containing missing values (`geom_scattermore()`).
## Warning: Removed 98598 rows containing missing values (`geom_scattermore()`).
## Warning: Removed 679 rows containing non-finite values (`stat_density()`).
## Warning: Removed 679 rows containing non-finite values (`stat_boxplot()`).
## Warning: Removed 21184 rows containing missing values (`geom_scattermore()`).
## Warning: Removed 38247 rows containing missing values (`geom_scattermore()`).
## Warning: Removed 262 rows containing non-finite values (`stat_density()`).
## Warning: Removed 262 rows containing non-finite values (`stat_boxplot()`).
## Warning: Removed 14473 rows containing missing values (`geom_scattermore()`).
## Warning: Removed 19104 rows containing missing values (`geom_scattermore()`).
## Warning: Removed 131 rows containing non-finite values (`stat_density()`).
## Warning: Removed 131 rows containing non-finite values (`stat_boxplot()`).
## Warning: Removed 1034 rows containing missing values (`geom_scattermore()`).
## Warning: Removed 5665 rows containing missing values (`geom_scattermore()`).
## Warning: Removed 38 rows containing non-finite values (`stat_density()`).
## Warning: Removed 38 rows containing non-finite values (`stat_boxplot()`).
## Warning: Removed 49 rows containing missing values (`geom_scattermore()`).
## Warning: Removed 4680 rows containing missing values (`geom_scattermore()`).
## Warning: Removed 32 rows containing non-finite values (`stat_density()`).
## Warning: Removed 32 rows containing non-finite values (`stat_boxplot()`).
## Warning: Removed 48 rows containing missing values (`geom_scattermore()`).
## Warning: Removed 4680 rows containing missing values (`geom_scattermore()`).
## Warning: Removed 32 rows containing non-finite values (`stat_density()`).
## Warning: Removed 32 rows containing non-finite values (`stat_boxplot()`).
## Warning: Removed 48 rows containing missing values (`geom_scattermore()`).
## Warning: Removed 4680 rows containing missing values (`geom_scattermore()`).
## Warning: Removed 32 rows containing non-finite values (`stat_density()`).
## Warning: Removed 32 rows containing non-finite values (`stat_boxplot()`).
## Warning: Removed 48 rows containing missing values (`geom_scattermore()`).
## Warning: Removed 4683 rows containing missing values (`geom_scattermore()`).
## Warning: Removed 32 rows containing non-finite values (`stat_density()`).
## Warning: Removed 32 rows containing non-finite values (`stat_boxplot()`).
## Warning: Removed 48 rows containing missing values (`geom_scattermore()`).
## Warning: Removed 4680 rows containing missing values (`geom_scattermore()`).
## Warning: Removed 32 rows containing non-finite values (`stat_density()`).
## Warning: Removed 32 rows containing non-finite values (`stat_boxplot()`).
## Warning: Removed 48 rows containing missing values (`geom_scattermore()`).
## Warning: Removed 4684 rows containing missing values (`geom_scattermore()`).
## Warning: Removed 32 rows containing non-finite values (`stat_density()`).
## Warning: Removed 32 rows containing non-finite values (`stat_boxplot()`).
## Warning: Removed 48 rows containing missing values (`geom_scattermore()`).
## Warning: Removed 4679 rows containing missing values (`geom_scattermore()`).
## Warning: Removed 32 rows containing non-finite values (`stat_density()`).
## Warning: Removed 32 rows containing non-finite values (`stat_boxplot()`).
## Warning: Removed 48 rows containing missing values (`geom_scattermore()`).
## Warning: Removed 4680 rows containing missing values (`geom_scattermore()`).
## Warning: Removed 32 rows containing non-finite values (`stat_density()`).
## Warning: Removed 32 rows containing non-finite values (`stat_boxplot()`).
## Warning: Removed 48 rows containing missing values (`geom_scattermore()`).
## Warning: Removed 4679 rows containing missing values (`geom_scattermore()`).
## Warning: Removed 32 rows containing non-finite values (`stat_density()`).
## Warning: Removed 32 rows containing non-finite values (`stat_boxplot()`).
## Warning: Removed 48 rows containing missing values (`geom_scattermore()`).
## Warning: Removed 48 rows containing missing values (`geom_scattermore()`).
## Warning: Removed 48 rows containing missing values (`geom_scattermore()`).
## Removed 48 rows containing missing values (`geom_scattermore()`).
## Warning: Removed 7583 rows containing missing values (`geom_scattermore()`).
## Warning: Removed 16402 rows containing missing values (`geom_scattermore()`).
## Warning: Removed 37233 rows containing missing values (`geom_scattermore()`).
## Warning: Removed 245 rows containing non-finite values (`stat_density()`).
## Warning: Removed 245 rows containing non-finite values (`stat_boxplot()`).
## Warning: Removed 10611 rows containing missing values (`geom_scattermore()`).
## Warning: Removed 31761 rows containing missing values (`geom_scattermore()`).
## Warning: Removed 212 rows containing non-finite values (`stat_density()`).
## Warning: Removed 212 rows containing non-finite values (`stat_boxplot()`).
## Warning: Removed 48 rows containing missing values (`geom_scattermore()`).
## Warning: Removed 4679 rows containing missing values (`geom_scattermore()`).
## Warning: Removed 32 rows containing non-finite values (`stat_density()`).
## Warning: Removed 32 rows containing non-finite values (`stat_boxplot()`).
## Warning: Removed 7583 rows containing missing values (`geom_scattermore()`).
## Warning: Removed 16402 rows containing missing values (`geom_scattermore()`).
## Warning: Removed 37277 rows containing missing values (`geom_scattermore()`).
## Warning: Removed 245 rows containing non-finite values (`stat_density()`).
## Warning: Removed 245 rows containing non-finite values (`stat_boxplot()`).
## Warning: Removed 10611 rows containing missing values (`geom_scattermore()`).
## Warning: Removed 31761 rows containing missing values (`geom_scattermore()`).
## Warning: Removed 212 rows containing non-finite values (`stat_density()`).
## Warning: Removed 212 rows containing non-finite values (`stat_boxplot()`).
## Warning: Removed 48 rows containing missing values (`geom_scattermore()`).
## Warning: Removed 4679 rows containing missing values (`geom_scattermore()`).
## Warning: Removed 32 rows containing non-finite values (`stat_density()`).
## Warning: Removed 32 rows containing non-finite values (`stat_boxplot()`).
## Warning: Removed 7583 rows containing missing values (`geom_scattermore()`).
## Warning: Removed 16402 rows containing missing values (`geom_scattermore()`).
## Warning: Removed 37458 rows containing missing values (`geom_scattermore()`).
## Warning: Removed 245 rows containing non-finite values (`stat_density()`).
## Warning: Removed 245 rows containing non-finite values (`stat_boxplot()`).
## Warning: Removed 7583 rows containing missing values (`geom_scattermore()`).
## Removed 16402 rows containing missing values (`geom_scattermore()`).
## Warning: Removed 37458 rows containing missing values (`geom_scattermore()`).
## Warning: Removed 245 rows containing non-finite values (`stat_density()`).
## Warning: Removed 245 rows containing non-finite values (`stat_boxplot()`).
## Warning: Removed 10611 rows containing missing values (`geom_scattermore()`).
## Warning: Removed 31968 rows containing missing values (`geom_scattermore()`).
## Warning: Removed 212 rows containing non-finite values (`stat_density()`).
## Warning: Removed 212 rows containing non-finite values (`stat_boxplot()`).
## Warning: Removed 10611 rows containing missing values (`geom_scattermore()`).
## Warning: Removed 31966 rows containing missing values (`geom_scattermore()`).
## Warning: Removed 212 rows containing non-finite values (`stat_density()`).
## Warning: Removed 212 rows containing non-finite values (`stat_boxplot()`).
## Warning: Removed 7581 rows containing missing values (`geom_scattermore()`).
## Warning: Removed 16489 rows containing missing values (`geom_scattermore()`).
## Warning: Removed 37371 rows containing missing values (`geom_scattermore()`).
## Warning: Removed 245 rows containing non-finite values (`stat_density()`).
## Warning: Removed 245 rows containing non-finite values (`stat_boxplot()`).
## Warning: Removed 37371 rows containing missing values (`geom_point()`).
## Warning: Removed 83 rows containing missing values (`geom_line()`).
## Warning: Removed 7578 rows containing missing values (`geom_scattermore()`).
## Warning: Removed 16411 rows containing missing values (`geom_scattermore()`).
## Warning: Removed 37251 rows containing missing values (`geom_scattermore()`).
## Warning: Removed 245 rows containing non-finite values (`stat_density()`).
## Warning: Removed 245 rows containing non-finite values (`stat_boxplot()`).
## Warning: Removed 7578 rows containing missing values (`geom_scattermore()`).
## Removed 16411 rows containing missing values (`geom_scattermore()`).
## Warning: Removed 37251 rows containing missing values (`geom_scattermore()`).
## Warning: Removed 245 rows containing non-finite values (`stat_density()`).
## Warning: Removed 245 rows containing non-finite values (`stat_boxplot()`).
## Warning: Removed 11715 rows containing missing values (`geom_scattermore()`).
## Warning: Removed 32871 rows containing missing values (`geom_scattermore()`).
## Warning: Removed 212 rows containing non-finite values (`stat_density()`).
## Warning: Removed 212 rows containing non-finite values (`stat_boxplot()`).
## Warning: Removed 11715 rows containing missing values (`geom_scattermore()`).
## Warning: Removed 32871 rows containing missing values (`geom_scattermore()`).
## Warning: Removed 212 rows containing non-finite values (`stat_density()`).
## Warning: Removed 212 rows containing non-finite values (`stat_boxplot()`).
## Warning: Removed 7583 rows containing missing values (`geom_scattermore()`).
## Warning: Removed 16402 rows containing missing values (`geom_scattermore()`).
## Warning: Removed 37233 rows containing missing values (`geom_scattermore()`).
## Warning: Removed 245 rows containing non-finite values (`stat_density()`).
## Warning: Removed 245 rows containing non-finite values (`stat_boxplot()`).
## Warning: Removed 7583 rows containing missing values (`geom_scattermore()`).
## Warning: Removed 16403 rows containing missing values (`geom_scattermore()`).
## Warning: Removed 37234 rows containing missing values (`geom_scattermore()`).
## Warning: Removed 245 rows containing non-finite values (`stat_density()`).
## Warning: Removed 245 rows containing non-finite values (`stat_boxplot()`).
## Warning: Removed 7583 rows containing missing values (`geom_scattermore()`).
## Removed 16403 rows containing missing values (`geom_scattermore()`).
## Warning: Removed 37234 rows containing missing values (`geom_scattermore()`).
## Warning: Removed 245 rows containing non-finite values (`stat_density()`).
## Warning: Removed 245 rows containing non-finite values (`stat_boxplot()`).
## Warning: Removed 10804 rows containing missing values (`geom_scattermore()`).
## Warning: Removed 31954 rows containing missing values (`geom_scattermore()`).
## Warning: Removed 212 rows containing non-finite values (`stat_density()`).
## Warning: Removed 212 rows containing non-finite values (`stat_boxplot()`).
## Warning: Removed 10804 rows containing missing values (`geom_scattermore()`).
## Warning: Removed 31954 rows containing missing values (`geom_scattermore()`).
## Warning: Removed 212 rows containing non-finite values (`stat_density()`).
## Warning: Removed 212 rows containing non-finite values (`stat_boxplot()`).
## Warning: Removed 10804 rows containing missing values (`geom_scattermore()`).
## Warning: Removed 31954 rows containing missing values (`geom_scattermore()`).
## Warning: Removed 212 rows containing non-finite values (`stat_density()`).
## Warning: Removed 212 rows containing non-finite values (`stat_boxplot()`).
## Warning: Removed 7583 rows containing missing values (`geom_scattermore()`).
## Warning: Removed 16404 rows containing missing values (`geom_scattermore()`).
## Warning: Removed 37235 rows containing missing values (`geom_scattermore()`).
## Warning: Removed 245 rows containing non-finite values (`stat_density()`).
## Warning: Removed 245 rows containing non-finite values (`stat_boxplot()`).
## Warning: Removed 7583 rows containing missing values (`geom_scattermore()`).
## Warning: Removed 16402 rows containing missing values (`geom_scattermore()`).
## Warning: Removed 37233 rows containing missing values (`geom_scattermore()`).
## Warning: Removed 245 rows containing non-finite values (`stat_density()`).
## Warning: Removed 245 rows containing non-finite values (`stat_boxplot()`).
## Warning: Removed 7583 rows containing missing values (`geom_scattermore()`).
## Removed 16402 rows containing missing values (`geom_scattermore()`).
## Warning: Removed 54056 rows containing missing values (`geom_scattermore()`).
## Warning: Removed 361 rows containing non-finite values (`stat_density()`).
## Warning: Removed 361 rows containing non-finite values (`stat_boxplot()`).
## Warning: Removed 10611 rows containing missing values (`geom_scattermore()`).
## Warning: Removed 31761 rows containing missing values (`geom_scattermore()`).
## Warning: Removed 212 rows containing non-finite values (`stat_density()`).
## Warning: Removed 212 rows containing non-finite values (`stat_boxplot()`).
## Warning: Removed 10611 rows containing missing values (`geom_scattermore()`).
## Warning: Removed 31761 rows containing missing values (`geom_scattermore()`).
## Warning: Removed 212 rows containing non-finite values (`stat_density()`).
## Warning: Removed 212 rows containing non-finite values (`stat_boxplot()`).
## Warning: Removed 10611 rows containing missing values (`geom_scattermore()`).
## Warning: Removed 31761 rows containing missing values (`geom_scattermore()`).
## Warning: Removed 212 rows containing non-finite values (`stat_density()`).
## Warning: Removed 212 rows containing non-finite values (`stat_boxplot()`).